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A Resolution Independent Nonrealistic Imaging System for Artistic Use 



Atsushi Kasao 

Graduate School of Information Science and 
Engineering, Tokyo Institute of Technology, 
2-12-1 Ookayama Meguro-ku, Tokyo, 152 Japan 

Abstract 

In multimedia environments, digital artwork is usually ex- 
hibited on many types of media with different sizes and reso- 
lutions. For this reason, it is desirable that artists create works 
of art in a resolution independent way. In this paper, we 
present an algorithm for resolution independent nonrealis- 
tic imaging. To realize resolution independent imaging, char- 
acteristics of all brush strokes are coded and saved in a char- 
acteristic table. From the characteristic table, images of any 
sized and resolution can be created. This characteristic table 
gives our algorithm the ability to create many types of artis- 
tic expression. Three different expressions are exhibited here. 
In particular the expression in which contours are drawn by 
pen and ink then colored by brush is , we believe, a new 
type of expression in computer artwork. Another feature of 
our algorithm is that the shape of a brush stroke is not set, 
but is adaptively changed. A less mechanical pattern which 
seems more natural to humans. As for esthetic evaluation, 
we do not have a proper evaluation method, but if an image 
created using this algorithm won an award for digital cre- 
ation, that would be an indication of the algorithm's suc- 
cess. 



1. Introduction 

Recently, the chance to see electronic art work through 
multimedia systems has been increasing. Because each me- 
dia usually has own resolution, it is better for artists to pro- 
duce art work in a resolution independent way. We are study- 
ing a way to produce a resolution independent automatic 
imaging system for use by artists. 

Many kinds of nonrealistic imaging techniques have been 
previously presented. Pen-and-ink illustration [1][2] and 
painterly rendering [3] [4] are two representative algorithms. 
In regard to pen-and-ink illustration, Salisbury et al. [1] pro- 
posed a way to produce a same density illustration for any 
magnification. The conventional painterly algorithm could 
also be made available in multimedia environments by 
changing the brush stroke size and reference photograph size 
in proportion to magnification. Combining the painterly al- 
gorithm and multiresolution images [5] is another possibil- 
ity. However, neither of these methods would work proper 
at high magnification because a magnified image is warped 
excessively. A better strategy is to change the imaging al- 
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gorithm itself to be independent of image size. 

Other than these problems, quality and variety of expres- 
sion have to be considered. Some nonrealistic rendering al- 
gorithms [3] [4] for animation and real-time processing are 
available, but we are aiming to produce an automatic nonre- 
alistic imaging algorithm for artistic use. 

In the history of art, many new techniques were invented 
and combined in one image. For example, Albrecht Durer 
drew "The citadel of Arco in the South Tyrol" [6] by water 
color and pen. Later Jean Honore Fragonard drew "Scene in 
a park" by pen-and-brush and brown ink, and watercolor 
wash. At the very least, imaging algorithm for artists must 
have the ability to produce many type of expressions. 

In the past, we interviewed various artists about what char- 
acteristics of a model they pay attention to when they sketch 
[7]. From these interviews, it was clear that artists grasp 
different characteristics according to their sketching styles. 
In an imaging algorithm having a variety of expressions, 
extracting many characteristics before creating an image is 
important. 

We propose a novel automatic nonrealistic imaging algo- 
rithm satisfying the above two characteristics (i.e. indepen- 
dence of image size and extraction of many characteristics 
before image creation). This algorithm consists of four parts. 
The first part is segmenting the image, extracting character- 
istics (average color and shape) from each segment and stor- 
ing these data into a characteristic table. The second part is 
arranging the characteristic table. The third part is creating 
brush strokes from the characteristic table only, and the fi- 
nal part is coloring the brush strokes. In this paper, we show 
that this algorithm can produce any size image and also can 
create a special look image painted by brush with contours 
drawn by pen and ink. 

The structure of this paper is as follows: section 2 gives 
an outline of our clustering algorithm used to produce the 
characteristic table; section 3 illustrates the basic algorithm 
for creating a comparatively realistic image; section 4 shows 
that this basic algorithm can produce any resolution images; 
section 5 illustrates and discusses how to produce a variety 
of expressions; and section 6 concludes with final remarks 
and a discussion of future work. 

2. Image segmentation 

In general speaking, a nonrealistic rendering image is gen- 
erated directly from a reference photograph. To produce a 



nonrealistic image only from a characteristic table instead 
of a reference photograph, the segmentation method which 
segments the photograph must have the ability to create such 
fine segments that one segment is considered one brush 
stroke. In addition, extracted characteristics from each seg- 
ment must be appropriate enough to produce many kinds of 
artistic images. Using the characteristic table, our first tar- 
get is to create a realistic rendered image similar to the refer- 
ence photograph. A realistic rendered image is defined as an 
image which consists of brush strokes but keeps important 
impressions of the reference photograph. As pixels in a brush 
stroke have the same color, the total amount of data is much 
less than the reference photograph. On the other hand, be- 
cause this algorithm is not just a lossy compression algo- 
rithm, the ability to create a variety of expressions is more 
important than compression ratio. The more effective the 
extracted characteristics, the smaller the number of segments 
required to produce a properly rendered image. The charac- 
teristics extraction process proceeds as follows: 

1 . An reference photograph is segmented into small seg- 
ments which have neighboring similar color pixels. This seg- 
mentation technique must create brush stroke like segments. 

2. The characteristics of each segment are entered into a 
characteristic table. 

In the past, we observed how painters sketch a model [7]. 
The results of these observations gave much useful infor- 
mation for creating our segmentation algorithm. 

The observations showed the following expression fea- 
tures: (a) Strokes on contours are very thin, (b) Strokes on 
details are fine. On the contrary, strokes on coarse areas are 
large, (c) In fine texture regions like hair, strokes usually do 
not express hair one by one, but draw an area of hair. The 
shape of such a stroke is usually stretched along the direc- 
tion of a texture, (d) If a non-texture area is surround by 
textured areas, shapes of strokes in the no-texture area are 
affected by shapes of strokes in the textured areas. These are 
not strict rules. To produce brush strokes as mentioned above, 
a 5-dimensional K-means clustering algorithm is adopted. 
The five dimensions consist of two position parameters and 
three color parameters, because a brush stroke must be a 
collection of pixels being near in position and in color space. 

Furthermore, to control brush strokes in accordance with 
above-mentioned rules, the 5-dimensional K-means cluster- 
ing algorithm must be revised to include information about 
direction and strength of a texture. In following section, we 
describe how to calculate the direction and strength of a tex- 
ture, and how to use this information. 

2.1. Notation 



b :b* 

db n : Edge strength of the nth pixel 
da n : Edge direction of the «th pixel 
Da.:. Texture direction of the y'th segment at the /th itera- 
tion 

Dd.y. Directionality strength of the y'th segment at the /th 
iteration 

Dc.y. Texture strength of the y'th segment at the /th itera- 
tion 

First, for pixels with color discontinuity, strength and edge 
direction of the color discontinuity need to be extracted from 
the reference photograph. The convolutions of the two masks, 
M S } and M 3 2 , shown in Figure 1, with the / component of 
the reference photograph G are performed respectively. Then, 

da and db are calculated using following equations: 



16 . -\,LL jc 

da n = — (tan ( )+— ) 

" 7t CC 2 
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db n = log£Z, 2 +CC 2 + l) 


(2) 


CC = M 3 .,*G 


(3) 


LI = M 3 . 2 * G 


(4) 



where * means convolution, and da n and db n are quantized 
and expressed by integers ranging from 0 to 15. The values 
of da n express the directions shown in Figure 2. 

2.2. Three texture factors extracted from texture in a 
segment 

The K-means algorithm is a method to iterate calculations 
of a distance between a pixel and the center of a segment 
successively until the shapes of segments do not change. Ac- 
cordingly, to give the segmentation algorithm the ability to 
utilize texture information, the distance function must be 
modified. 

Generally speaking, to perceive some texture, a single pixel 
is not enough. A small area consisting of some pixels is re- 
quired. We will show how to extract texture information from 
da n and db n of a segment using the following procedures. 

Let /' be the index of iteration. First, Af is defined as the 

0,1,/ 

number of pixels which meet the conditions da n = 0 and db n 
> 1 in the yth segment at the /th iteration. In the same man- 
ner, N^j (k = 1~1 5) are defined. Then, N l6t . is defined as the 
numberof pixels having no direction (i.e. tne number of pix- 
els meeting the condition db n <\ in the jth segment at the ith 



We selected the CIELAB color space, because this color 
space is fit for human vision and the formula for a color 
difference calculation is easy to define . Here, we define the 
symbols used in the following sections: 
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iteration). We have found that the threshold db<\ works well. 
The maximum value from N QJJ to Af, v . is chosen. If this value 
is larger than N l6i J\0, Da., is set as trie representative direc- 
tion of the yth segment. On the other hand, if the maximum 
number is equal to or smaller than N l6 .J\0, the number 16 is 
assigned to Da.., which means that the yth segment has no 
texture. 

We introduce the measure Dd. . which indicates how clearly 
Da. . represents the direction of a segment. It is assumed that 
a human is able to discriminate at most two texture direc- 
tions at a time and if there is a second direction, it must dif- 
fer from the first direction by about nil. Therefore, Dd Q is 
expressed by the following equations: 

^ = ^K-<J/(^ ( /<,,) (5) 
= [N^N^N^yz (6) 

where k 2 is a real constant ranging from 0 to 1 . 

If the result of the calculation of Da. . is less than 0, Da. . 
must be increased by 1 6 . On the other fiand, if the result of 
calculation of Da., is more than 1 5, Da., must be reduced by 
16. The threshold, N l6 .J\Q, is chosen from previous trial ex- 
periments. 

These two characteristics of texture, Da tJ and Dd.., are not 
enough to enable a segmentation algorithm to use texture. 
As mentioned before, when painters draw fine textures, they 
are inclined to neglect fine detail and boldly paint the fine 
texture with broad strokes. However, the bold strokes are 
usually painted along the direction of the texture. To pro- 
duce an algorithm which can simulate the features described 
above, strength of the texture, Dc.., is necessary to control 
the parameters of the segmentation. Dc.. is calculated by the 
following equation: 

15 

Dc iJ = k l ^ d N kiJ /N iJ (7) 

where k x is a positive constant. 
These three texture factors {Da . JDd. ,Dc. ) are used for con- 

V V V 

trolling the distance function in this algorithm. 

23. K-Means Algorithm with Three Factors of Tex- 
ture 

As was mentioned in the previous section, this algorithm 
is one of methods used to iterate the calculation of the dis- 
tance function until the result converges. This segmentation 
algorithm is executed according to the following steps: 
Step 1. Initial arrangement of segment centers 

Let K be the initial number of segments. A total of K rect- 
angles are put side by side on an reference photograph G. 
These rectangles are regarded as the initial segments of G, 
and all above mentioned characteristics in segments are cal- 



culated. The characteristics of the jth segment at the ith it- 
eration (C.J) consist of the center of gravity (X. ,7^), the num- 
ber of pixels (TV. .), the average color (L ij9 A iJt B.J), the direc- 
tion of texture (Da. ), strength of directionality (Dd. ) and 
strength of texture (Dc.J). In short, C Q consist of (X^Y^, N.., 
L.., A.., B. J9 Da (J , Dd {J , Dc.J). Initial segment characteristics 
(i.e. /' = 0) C 0J are calculated and written to the characteristic 
table. Similar calculations are repeated at each iterations until 
a stopping condition is satisfied. Assuming that C. are be- 
ing calculated at the ith iteration, we will illustrate follow- 
ing steps . 

Step 2. Judging a segment 
Pixels are chosen one by one. Then, the weighted Euclid- 
ean distance, H . , between C. . and the chosen nth pixel is 
calculated by the Equations (8)~(12). First, the distance be- 
tween the center of gravity and the pixel position, and the 
color difference between C . and nth pixel are calculated. 



(io) 

%.« = (. A ,j-<*n) (11) 
%-QtJ- b *) (12) 



Second, the weighted distance between the center of grav- 
ity and the pixel's position is calculated, taking the texture 
into account. 

Ax'.- =F(Ax.. ,Ay.. ,Da- ,Dd . ,Dc )Ax- . (13) 
Ay'.. = F(Ajc.. ,Av.. ^Da.^Dd.^Dc^Ay.. (14) 

where F(») is the function to control the shape and size of 




Figure 2: The directions from 0 to 15 



segments. 
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Step 3. Recording to the characteristic table 

C in all newly calculated segments are written over C 1 . 
in the characteristic table. 

Step 4. Checking the stopping condition 

The discrepancy H\. between C. and C. is calculated by 
the following equations: 



x{l+Dd 



J Ax.. 2 71 
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(15) 
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F(A*.. ,Ay.. ,Da .,Dd- ,Dc ^)=\ J £>c.. 



x{l+D</..(16- 



Da,.. -(tan ( J ' )+ — ) — 



/4-1)} (16) 



These equations are combined, and H. Jn is defined as 

=kAx'.. 2 +kAy' - 2 

+k,Alun 2 +K^U,n 2+k ^b Un 2 (17) 



where k ,k ,k„k Jc. are constants. 

The distance H. between all segments and the nth pixel 
are calculated and* the nth pixel is included in the yth seg- 
ment with the smallest value of H. . . Figure 3 shows the 

tj.n ° 

relationship between the vector from the center of gravity 
(X.^Y. ) to the nth pixel and the direction of texture in a seg- 
ment t>a. .. If the direction of this vector comes close to Da 
the value of H. . is decreased. On the other hand, if the di- 

y.n ' 

rection differs from Da . by 7t/2, the value of H. . . is increased. 
As a result, the yth segment includes many pixels in the di- 
rections close to Da.. or Da. tJ + rc, but includes few pixels in 
the other directions. 

Furthermore, if almost all pixels in the yth segment have 
similar values to da and satisfy the condition db >1 , then 

n J n 9 

Dd tJ ~ k 2 , and the shape of the segment is stretched to be a 
narrow ellipse. On the contrary, if many pixels in the yth 
segment satisfy only the condition, db n >\, regardless of the 
direction, then Dc u ~k y Eventually, the size of the yth seg- 
ment becomes small and the contour of a segment becomes 
simple. This fact is because of the following reason: if the 
value of Dc.. is close to k ]9 ¥(•) becomes large. These calcu- 
lations are executed successively for every segment. 





(18) 
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(20) 


A A,i + \J=^i + lJ- A i.j) 


(21) 


A BJ+\J = ®i+\j-Bij) 


(22) 


and 





A 2 +k' A 2 



(23) 



The above steps are repeated until H f . . has a smaller value 
than a previously chosen value (Crf,). have found that a 
value of Cd x = 1 works well. When this iterative process 
stops, the segmentation is completed. 

As this new 5-dimensional K-means clustering algorithm 
is a very time consuming process, we chose to divide the 
reference photograph into small rectangular regions and ap- 
ply the algorithm to each region separately. Figure 4 shows 
the rectangular areas consisting of a core area and overlap- 
ping areas [8]. The overlapping areas surround the core area. 
The new clustering algorithm is applied to each rectangular 
area. Without overlapping areas, each small rectangular area 
would be clustered independently. That would cause a dis- 
continuity of a segment on the border. However, if there are 
overlapping areas, these are clustered several times. This 
procedure decreases the defects of discontinuity caused by 
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Figure 3: Positional relationship between the pixel n 
and the center of gravity of the yth segment (X. ) 



separately applying the K-means clustering algorithm to 
small areas. 

This explanation outlines our new clustering algorithm. 
Regarding this algorithm, deciding a suitable number of seg- 
ments at the beginning is difficult. However, this algorithm 
can segment each small area according to different condi- 
tions. The conditions are changed while this algorithm is 
repeatedly apply to the same small rectangular area. Accord- 
ingly, the number of segments can be increased until the 
maximum color deviation of pixels in a segment is within a 
certain value (Cd 2 ). We have found that a value of Cd 2 = 20 
works well. This segment increasing process efficiently pro- 
duces enough segments for realistic imaging. Finally, when 
whole process is finished, the group of characteristics (C.) 
is written to the corresponding cell (C.) in the characteristic 
table. In the following sections, we will call this algorithm 

the adaptive K-means method. 

3. The basic algorithm for a realistic image 

3.1. Additional characteristics 

As mentioned in the previous section, C. consists of (X., 

Y., N., L.,A., B, Da., Dd., Dc). C is not enough to create 

j j j j j j j j j 

brush strokes for a realistic image, because it does not con- 
tain segment shape information. Assuming that each brush 
strokes is an ellipse, let D be the direction of the long axis 
and let H. be the oblate rate. These two value are added to 
C D and H. are calculated from the deviations of pixels in 
the segment. First, the dispersion in relation to the x-direc- 
tional deviation from the segment center is defined by dx., 
the jy-directional dispersion is defined by dy j9 and the co- 
dispersion is defined by k.. Furthermore, let the first and sec- 
ond primal element be £ } . and L 2J respectively. These two 
values are calculated from the above values. D. and H. are 

then obtained as followers: 



-W i 

£>=tan' l ( ^)/7C+~ 

W 2j 2 



(24) 
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Fig. 4: An example of an image divided 
into many small areas. 



Hj=(L xr L 2j )l(L XJ +L 2J ) 
where 



(25) 



Jkj 2 -HL ir dxj)* 



If W + (^;A) 2 0* 



W XJ 0 



If jk*+{L XJ -dxy 0, (26) 



*2j V 1 \ w u ( 2 7) 
Finally, D p H. are added to C in the characteristic table 

3.2. The basic imaging algorithm 

The reference photograph and the results of segmentation 
are shown in Figures 5(a) and 5(b) respectively. Figure 5(c) 
is produced by painting over every segment by the average 
color of that segment This image resembles the reference 
photograph and the shapes of almost all segments are el- 
lipses, similar to brush strokes. Therefore, the adaptive K- 
means method is considered to work well. 

In order to reproduce the shapes of segments in Figure 
5(c) from only the characteristics of the characteristic table, 
we need a reproduction process. The following steps show 
this reproduction process. 

[Step 1] Assign the pixel numbers 

This method can produce an image of any size. For sim- 
plicity, we will make the image the same size as the refer- 
ence photograph. 

In the first row of the image, pixels are assigned serial 
numbers from the left side to the right side. In the next row, 
pixels are assigned serial numbers from the right side to the 
left side. In this manner, the pixels in odd rows of the refer- 
ence photograph are assigned serial numbers from left to 
right, and pixels in even rows are assigned serial numbers 
from right to left. This way of numbering distributes all se- 
rial numbers on the image side by side. 

[Step 2] Create the neighboring segments table 

For every segment, the nearest 30 segments are chosen 
and these segments are written to the neighboring segments 
table shown in Table 1 in order of shortest distance. 

[Step 3] Creating the brush strokes 

A brush stroke j on a new image is created from a corre- 
sponding segment j on a segmented image. In the same way 
as the previously shown segmentation, the position of a brush 
stroke is indicated on the new image by a corresponding seg- 
ment number. Because pixel n is next to pixel if pixel n- 
1 is a member of brush stroke j 9 pixel n must be in one of the 
nearest 30 segments of segment j. 

Accordingly, after calculating the distances between the 
nearest 30 neighbors and pixel n by equation (28), the pixel 
n is chosen as a member of the brush stroke which has the 

smallest M J (i.e. the shortest distance) . 
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Figure 5: Brush strokes made from an photo image: (a) 
the original image, (b) the segmented image, (c) an im- 
age colored by the average colors of segments in (b), 
with a magnified insert, (d) brush strokes made from a 
characteristic table, (e) an image painted by the average 
colors of brush stokes in (d), with a magnified insert, 
(f) brush strokes made from a revised characteristic table, 
and (g) an image painted by average the colors of brush 
stokes in (f), with a magnified insert. 
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As can be seen, in equation (28), M. is inversely propor- 
tional to N. For this reason, a pixel is apt to be a member of 
the segment j with a large N.. When H * 0, pixels in the 
direction D. from the center of* the segment j are more likely 
to be a member of the segment j than pixels in other direc- 
tions. The above method makes brush strokes similar to the 
segments on the just segmented image. When this proce- 
dure is performed for every pixel, Figure 5(d) is produced. 

[Step 4] Coloring process 

This step is to paint each brush stroke with its average 
color. Figure 5(e) is produced by painting over every brush 
stroke of Figure 5(d) by the average color of that segment. 

3.3. The propagation algorithm of characteristics 

Figure 5(e) is similar to Figure 5(c) in that both of the 
figures have the previously mentioned desired expression 
features (a) ~ (c) (see the introduction of section 2). How- 
ever, their brush stroke gives a mechanical feeling because 
brush strokes on the area without texture are simple rect- 
angles. In other word, their brush strokes do not have the 
expression feature (d). The expression feature (d) is that if 

Table 1 : The Neighbor Segments Table 

Segment the nearest 30 segments 
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the non-texture area is surround by texture areas, the shape 
of the stroke on the non-texture area is affected by the shape 
of strokes on the surrounding texture area. We will show 
how to add this feature to Figure 5(e). From experience, if 
H. is larger than 0.7, the shape of the brush stroke resembles 
a real brush stroke. If H. is not larger than 0.7, the top 10 
largest H k from the segments in the neighbor segments table 
are chosen. Then, H k which meet the condition, H k > H f are 
averaged, and this value is substituted for H.. In the same 
way, D is replaced for the average. If this process is repeated 
many times, the influence of a brush stroke with large H. 
propagates to other brush strokes far away. From the results 
of previous experiments, we decide to iterate this process 
100 times. Brush strokes created by using this revised char- 
acteristic table are shown in Figure 5(f), and Figure 5(g) is 
the image which is painted using the average color of each 
brush stroke in Figure 5(f). It is recognized that the brush 
strokes of Figure 5(f) are much more proper for painterly 
imaging than Figure 5(d). 

4. Resolution Independent Imaging 

In Figure 6, four different-sized images are shown. All 
images were produced by our imaging algorithm. Because 
this algorithm produces the painterly image using only the 
characteristic table, however, high resolution or any large- 
sized images can be created by using the same algorithm. In 
Figure 6, all images consist of 1218x815 pixels. Figure 6(a) 
is the whole image, Figure 6(b) is part of a 2436x1 630 (2x) 
image, Figure 6(c) is part of a 3654x2445 (3x) image, and 
figure 6(d) is part of a 4872x3260 (4x) image. None of these 
image show any jagged edges. 

5. Brush expression with contours by pen and 
ink and a variety of other expressions 

In previous sections, It was shown that this algorithm had 
the ability to produce any size painterly image using only 
the characteristic table. However, if this algorithm could not 
create a variety of expressions, it would be useless. This al- 
gorithm has mainly three ways to add different expressions 
to the basic painterly image (i.e. Figure 5 (e)). One way is to 
modify step 3 of the basic imaging algorithm. Another one 
is in step 4, and the last place is in relation to the propaga- 
tion algorithm of the characteristic table. The last method is 
used for propagating conspicuous characteristics to surround- 
ing segments. In Figure 5(e) shapes of brush strokes were 
propagated, but color can also be propagated. We will illus- 
trate two new expressions by modifying step 2 and step 3, 
and the previously mentioned hybrid expression painted by 
brush with contours drawn by pen and ink. 

5.1 Expression with impressionist-like brush strokes 

The shape of a brush stroke varies by modifying the equa- 
tion (28). To make the shape of the brush stroke fine and 
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Figure 6: The results of resolution independent imaging 




Figure 7: Expression with impressionist-like brush strokes. 




Figure 8: Special expression with color gradation within brush strokes. 
('97 Toray Creation Awards, Exereses Award) 




Figure 9: An impressionist style painting with contours by pen and ink. 



long, equations (28) ~ (30) are replaced by equations (31) ~ 
(33). 
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The color of brush strokes can be changed by varying L , 
A. t and B. of the characteristic table by a random number 
ranging between ±20. This color modification can add to the 
image a touch of impressionist painting. The result is shown 
in Figure 7. 



5.2 Special expression with color gradation within 
brush strokes 



This algorithm can create two types of color gradation in 
brush strokes. The shape of a brush stroke of Figure 8 is the 
same as that of Figure 5(g). This image was produced with 
the following conditions: if the value of H. is under 0.9, the 
segment j is colored by a star shaped gradation. On the other 
hand, if the value of H is more than 0.9, this brush stroke is 
colored by gradation along the direction of the normal to Z).. 

5.3 Brush expression with contours by pen and ink 

As was previously mentioned, besides creating a variety 
of expressions, expressing different techniques on one art- 
work at a time is important. Figure 9 seems to have strokes 
painted by brush with pastel colors and contours by pen and 
ink. In this image a contour line is a segment which meets 
the condition (H. > 0.85) being made very slender and col- 
ored black. When H < 0.85, equation (28) and (29) are used 
for producing the usual brush strokes. On the other hand, 
when H > 0.85, (28) and (29) are substituted by following 

equations to turn the segments into contour lines. 
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